United States Government Accountability Office 



GAO 



Report to Congressional Requesters 



HEAD START 
A More 

Comprehensive Risk 
Management Strategy 
and Data 

Improvements Could 
Further Strengthen 
Program Oversight 




GAO 

Accountability * Integrity * Reliability 



GAO-08-221 



February 2008 



A GAO 

Accountability- Integrity Reliability 

Highlights 

Highlights of GAO-08-221 , a report to 
congressional requesters 



Why GAO Did This Study 

In February 2005, GAO issued a 
report that raised concerns about 
the effectiveness of the Department 
of Health and Human Services 
(HHS) Administration for Children 
and Families’ (ACF) oversight of 
about 1,600 local organizations that 
receive nearly $7 billion in Head 
Start grants. GAO was asked to 
report on (1) ACF’s progress in 
conducting a risk assessment of the 
Head Start program and ensuring 
the accuracy and reliability of data 
from its annual Program 
Information Report (PIR) survey of 
grantees, (2) efforts to improve on- 
site monitoring of grantees, and (3) 
how data are used to improve 
oversight and help grantees meet 
program standards. For this report, 
GAO surveyed a nationally 
representative sample of Head 
Start program directors and 
interviewed ACF officials. GAO 
also reviewed ACF studies on the 
validity of PIR data and conducted 
tests of data from the 2006 PIR 
database. 



What GAO Recommends 



GAO recommends that ACF 
develop a more strategic approach 
to assessing risks to the Head Start 
program, expand its efforts to 
collect data on and estimate 
improper payments, improve the 
accuracy of data from its annual 
PIR survey, and develop clear 
criteria for providing additional 
assistance for high-risk grantees. 
ACF agreed with the latter two 
recommendations and emphasized 
progress already made toward 
developing a comprehensive risk 
assessment process and reducing 
improper payments. 

To view the full product, including the scope 
and methodology, click on GAO-08-221. 

For more information, contact Cornelia Ashby 
at (202) 512-7215 or ashbyc@gao.gov. 
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What GAO Found 

ACF has not undertaken a comprehensive assessment of risks that may limit 
Head Start’s ability to meet federal program objectives, despite GAO’s 2005 
recommendation, and little progress has been made to ensure that the data 
from its annual PIR survey of grantees, which could facilitate such an 
assessment, are reliable. To conduct a comprehensive risk assessment, ACF 
needs to identify external and internal risks, estimate their significance, and 
decide how to best manage them. While ACF says it is working to establish 
two systems to address programwide risk, our analysis suggests that these 
systems fall short of that goal. The first system, by which ACF assesses 
grantees before providing new funds each year, only assesses risk posed to 
the program by poorly performing grantees and does not allow for a broader 
assessment of other sources of risk, such as improper payments to 
contractors. The second system, a new, integrated management information 
system, has been in development for over 4 years and it is not clear how it will 
facilitate a more comprehensive risk assessment for the Head Start program. 
Both initiatives depend, in part, on data from the annual PIR survey of 
grantees, which have been found to be unreliable. 

ACF has taken steps to improve oversight of Head Start grantees by 
implementing a more rigorous process for certifying reviewers who conduct 
on-site monitoring visits, implementing new processes to improve the 
consistency of reviews, and working to establish a system for evaluating 
reviews on an ongoing basis. Now, ACF verifies reviewers’ qualifications and 
requires them to pass online tests in writing and computer literacy. Reviewers 
must also complete ongoing training and are evaluated by their peers at the 
end of each review. ACF has also taken a number of steps to improve the 
consistency and objectivity of reviews, including developing a Web-based data 
collection tool to facilitate information gathering, assigning review team 
leaders from outside the grantee’s home region to increase independence, and 
centralizing the review and preparation of monitoring reports. ACF is also 
working to establish an ongoing system for evaluating its on-site review 
process. 

ACF uses data to track grantee performance and target assistance to 
underperforming grantees, but weaknesses may have hindered these efforts to 
improve grantee performance. For example, ACF does not have clear criteria 
for determining which grantees need additional oversight as part of its 
refunding analysis. Such decisions are made on an ad-hoc basis, which may 
result in grantees with similar problems receiving different levels of oversight. 
Moreover, prior to the December 2007 reauthorization of Head Start, ACF was 
limited in its ability to increase competition for grants to replace 
underperforming grantees. Under the new law, ACF will have more flexibility 
to open competition for Head Start grants to other prospective grantees when 
current grantees fail to deliver high-quality, comprehensive Head Start 
programs. 
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Abbreviations 



ACF Administration for Children and Families 

HSES Head Start Enterprise System 

HHS Department of Health and Human Services 

OMB Office of Management and Budget 

PIR Program Information Report 

T/TA Training and technical assistance 
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Accountability * Integrity * Reliability 



United States Government Accountability Office 
Washington, DC 20548 



February 12, 2008 
Congressional Requesters 

In February 2005, we issued a report that raised concerns about the 
effectiveness of oversight of the Head Start program. We made a number 
of recommendations to improve the Department of Health and Human 
Services (HHS) Administration for Children and Families’ (ACF) oversight 
of local organizations that receive Head Start program grants and the 
grantees’ financial management. 1 Head Start is one of the largest federal 
early childhood programs. In fiscal year 2007, Head Start provided nearly 
$7 billion to 1,604 local organizations that provide a wide range of services 
to over 900,000 primarily low-income children, from birth to age 5, and 
their families. These services are aimed at improving the social 
competence, learning skills, and health and nutrition status of low-income 
children so that they can begin school ready to learn. ACF spent $25.6 
million in fiscal year 2006 to conduct its oversight activities. 

Since our 2005 report, ACF has implemented a number of changes 
designed to improve its monitoring and other oversight of grantees. 
However, policymakers have raised new concerns about the scope of 
these changes and whether they are likely to strengthen the performance 
of Head Start grantees, suggesting a need to reassess ACF’s monitoring 
and oversight activities. To respond to your request for more information 
about ACF’s monitoring and oversight of the Head Start program, we 
examined: 

1. ACF’s progress in conducting a risk assessment of the Head Start 
program and ensuring the accuracy and reliability of data from its 
annual Program Information Report (PIR) survey of grantees, 

2. ACF’s efforts to improve its on-site monitoring processes, and 

3. ACF’s use of data to improve oversight and help grantees meet 
performance standards. 



1 GAO, Head Start: Comprehensive Approach to Identifying and Addressing Risks Could 
Help Prevent Grantee Financial Management Weaknesses, GAO-05-176 (Washington, D.C.: 
Feb. 28, 2005). 
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To address these objectives, we visited and interviewed Office of Head 
Start and ACF officials and their staff in Washington, D.C. We interviewed 
staff from all of ACF’s 10 regional offices and reviewed relevant 
documentation from each of these offices. We administered a Web-based 
survey to a nationally-representative sample of Head Start and Early Head 
Start program directors. 2 We selected our sample of directors from the 
population of directors whose programs were the subject of ACF’s on-site 
reviews from October 2005 through March 2007 using a newly revised on- 
site review process. In total, we surveyed 329 program directors, asking 
them about their experiences during the most recent on-site reviews, and 
their views about the changes to the process. Of the 329 program 
directors, 261 responded — for a response rate of 79 percent. Throughout 
this report, when we refer to our survey results, we use the terms 
“program directors” and “grantees” interchangeably. Finally, we tested the 
reliability of two of ACF’s administrative databases — the first contains 
data on the results of each grantee’s on-site review, and the second 
contains data from ACF’s annual PIR survey of grantees. To assess the 
reliability of on-site review data, we relied on our 2005 assessment of the 
reliability of these data, performed additional electronic tests of data 
elements used for an analysis of the extent of repeat on-site review 
findings, and reviewed new information about the database. Based on our 
previous and updated assessments, we find the on-site review data used 
for our analysis of on-site review findings to be sufficient for the purpose 
of this report. However, we identified concerns about the overall reliability 
of data from ACF’s annual PIR survey of grantees, which we discuss more 
fully in this report. To further address objective 1, we reviewed a recent 
ACF study on the validity of PIR survey data. To further address objective 
2, we met with the contractor responsible for coordinating the on-site 
review process and tracking on-site review results, and representatives 
from the National Head Start Association. We also examined patterns of 
on-site review results in areas related to program governance, record- 
keeping and reporting, and fiscal management for all programs reviewed 
in both fiscal years 2003 and 2006 to assess the extent of repeat findings, 
and obtained the results of a study conducted by ACF to evaluate its on- 
site review process. For additional details about our scope and 
methodology, see appendix I. Our work was conducted from February 



2 Head Start is authorized to serve children at any age prior to compulsory school 
attendance. The program was originally aimed at 3- to 5-year-olds. Early Head Start, begun 
in 1994, focuses on serving children from birth to 3 years of age and pregnant women. For 
the purposes of this report on program oversight, when we discuss the Head Start program, 
we are referring to both Head Start and Early Head Start. 
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Results in Brief 



2007 through December 2007 in accordance with generally accepted 
government auditing standards. 



ACF has not undertaken a comprehensive assessment of risks that may 
limit Head Start’s ability to meet federal program objectives, despite our 
2005 recommendation, and little progress has been made in ensuring that 
the data from its annual PIR survey of grantees, which could facilitate 
such an assessment, are reliable. Risk assessment is a key component of 
internal control that can provide the basis for determining what actions 
need to be taken to address problems that may limit an agency’s progress 
toward meeting federal program objectives. For the Head Start program, 
this would include a formal, ongoing process of identifying external and 
internal risks, estimating their significance, and deciding how to best 
manage them. While ACF has two systems in development to address 
programwide risk, our analysis suggests that these two systems fall short 
of that goal. First, ACF has a new system for assessing individual grantees 
each year before deciding whether to refund, or renew, their grants. This 
system partly addresses our 2005 recommendation that ACF produce a 
comprehensive risk assessment of the Head Start program. However, this 
refunding analysis system only assesses risk posed to the program by 
poorly performing grantees; it does not allow for a broader assessment of 
other sources of risk, such as those we previously identified, including 
limited efforts to assess regional office staff adherence to ACF policies 
and procedures and improper payments to contractors. Improper 
payments continue to be an issue for ACF. For example, in April 2007, the 
HHS Inspector General reported that 5 percent of Head Start slots that 
were paid for in 2006 had not been filled. Although ACF collects data on 
improper payments to grantees that enrolled too many children from 
families that did not meet the program’s income eligibility requirement, 
ACF officials stated that they do not have plans to track other types of 
improper payments. The second process that ACF is developing is a new, 
integrated management information system, known as the Head Start 
Enterprise System (HSES). However, HSES has been in development for 
over 4 years and it is not clear when or how it will facilitate a more 
comprehensive risk assessment for the Head Start program. Furthermore, 
both the refunding analysis system and the HSES depend, in part, on data 
from the annual PIR survey of grantees, which several studies have found 
to be unreliable. These studies suggest that PIR data are unreliable, partly, 
because of the length and complexity of the survey, which includes over 
130 questions that are sometimes difficult for grantees to answer. ACF 
does not independently verify the accuracy of data submitted by grantees. 
Despite its widespread use in providing the public with information on the 
national Head Start program and in overseeing grantees, our analysis also 
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raises significant concerns about the reliability of the data from this 
survey. 

ACF has taken steps to improve its oversight of Head Start grantees by 
implementing a more rigorous process for certifying reviewers who 
conduct on-site monitoring visits, implementing new processes to improve 
the consistency of reviews, and working to establish a system for 
evaluating reviews on an ongoing basis. Now, ACF verifies all potential 
reviewers’ qualifications and requires them to pass online tests in writing 
and computer literacy. Reviewers must also complete ongoing training and 
are evaluated by their peers at the end of each review. ACF has also taken 
a number of steps to improve the consistency and objectivity of reviews. 
For example, ACF has developed a Web-based data collection tool that 
provides a more structured way of gathering evidence on grantee 
performance. To help ensure objectivity of review teams, they are now led 
by program specialists from outside the grantees’ home regions who are 
directly accountable to the central Office of Head Start. Also, to improve 
on-site monitoring reports, review findings are now reviewed by central 
office staff for consistency. Based on the results of our survey of Head 
Start and Early Head Start program directors, grantees have mixed views 
about the revised on-site monitoring procedures. For example, although 
most grantees who were reviewed under the revised procedures were very 
or somewhat satisfied with how their most recent on-site review was 
conducted, they were almost evenly split over whether having a program 
specialist from outside their home region lead their review had a positive 
or negative effect on the review process. In 2005, we found evidence that 
some reviewers may not have followed on-site review guidelines and, as a 
result, some grantees were not reviewed as rigorously as others. In 2006, 
ACF evaluated its on-site reviews to determine whether it was followed 
consistently by different teams of reviewers and the extent to which the 
review process helped reviewers identify serious program problems. ACF 
found that, although review teams did not consistently identify the same 
program problems for the same grantees, none of the teams failed to 
identify serious program problems. ACF is working to establish an ongoing 
system for evaluating its on-site review process. 

ACF uses data to track grantee performance and to target assistance to 
underperforming grantees, but weaknesses in the extent to which 
performance data are used to guide oversight may have hindered this 
effort to improve grantee performance. ACF uses the results of its data 
analysis to target resources from its training and technical assistance 
network to help underperforming grantees address program weaknesses 
through workshops, on-site assistance, or other specialized training. 
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